Where Are My Intelligent Assistant's Mistakes? A Systematic Testing Approach
نویسندگان
چکیده
Intelligent assistants are handling increasingly critical tasks, but until now, end users have had no way to systematically assess where their assistants make mistakes. For some intelligent assistants, this is a serious problem: if the assistant is doing work that is important, such as assisting with qualitative research or monitoring an elderly parent’s safety, the user may pay a high cost for unnoticed mistakes. This paper addresses the problem with WYSIWYT/ML (What You See Is What You Test for Machine Learning), a human/computer partnership that enables end users to systematically test intelligent assistants. Our empirical evaluation shows that WYSIWYT/ML helped end users find assistants’ mistakes significantly more effectively than ad hoc testing. Not only did it allow users to assess an assistant’s work on an average of 117 predictions in only 10 minutes, it also scaled to a much larger data set, assessing an assistant’s work on 623 out of 1,448 predictions using only the users’ original 10 minutes’ testing effort.
منابع مشابه
I-36: Preimplantation Genetic Diagnosis - Where Have We Been and Where Are We Going
Preimplantation genetic diagnosis (PGD) is now considered routine in IVF laboratories with micromanipulation capability and access to genetic diagnostic services. The past two decades have witnessed a dramatic increase in the use of PGD, the number of cycles performed, and the indications for which PGD has been used. This increase has been mirrored by a slow, but steady, increase in the range o...
متن کاملClassifying and Detecting Plan-Based Misconceptions for Robust Plan Recognition
My Ph.D. dissertation (Calistri 1990) extends traditional methods of plan recognition to handle situations in which agents have flawed plans.1 This extension involves solving two problems: determining what sorts of mistakes people make when they reason about plans and figuring out how to recognize these mistakes when they occur. I have developed a complete classification of plan-based misconcep...
متن کاملAbbas Kiarostami, Family Film and the Techno-cultural Processes of Transcultural Viewing
In this paper Abbas Kiarostami's films for children are discussed from the perspective of a cognitive studies approach. The crux of the argument is that the visual elements of film are essentially metonymic. Where is My Friend’s Home? has a quest-script instantiated by means of four components drawn on Brown and Babbington. Transcultural viewing is enabled by techno-cu...
متن کاملAn Intelligent System’s Approach for Revitalization of Brown Fields using only Production Rate Data
State-of-the-art data analysis in production allows engineers to characterize reservoirs using production data. This saves companies large sums that should otherwise be spend on well testing and reservoir simulation and modeling. There are two shortcomings with today’s production data analysis: It needs bottom-hole or well-head pressure data in addition to data for rating reservoirs’ characteri...
متن کاملIntelligent Virtual Assistant's Impact on Technical Proficiency within Virtual Teams
Information-systemsdevelopmentcontinuestobeadifficultprocess,particularlyforvirtualteams thatdonothavetheluxuryofmeetingface-to-face.Theresearchliteratureonthistopicreinforces thispoint: thegreaterpartofdatabasesystemsdevelopmentprojectsends in failure.Theuseof virtualteamstocompleteprojectsfurthercompoundsthesefailures.However,recentdevelo...
متن کامل